Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-3116: [Plasma] Add "ls" to object store #2470

Closed
wants to merge 22 commits into from

Conversation

pcmoritz
Copy link
Contributor

@pcmoritz pcmoritz commented Aug 25, 2018

This adds plasma_client.list to the plasma client API.

It can be used like so:

  import pyarrow.plasma as plasma
  import time

  client = plasma.connect("/tmp/plasma", "", 0)

  client.put("hello, world")
  # Sleep a little so we get different creation times
  time.sleep(2)
  client.put("another object")
  # Create an object that is not sealed yet
  object_id = plasma.ObjectID.from_random()
  client.create(object_id, 100)
  print(client.list())

  >>> {ObjectID(4cba8f80c54c6d265b46c2cdfcee6e32348b12be): {'construct_duration': 0,
  >>>  'create_time': 1535223642,
  >>>  'data_size': 460,
  >>>  'metadata_size': 0,
  >>>  'ref_count': 0,
  >>>  'state': 'sealed'},
  >>> ObjectID(a7598230b0c26464c9d9c99ae14773ee81485428): {'construct_duration': 0,
  >>>  'create_time': 1535223644,
  >>>  'data_size': 460,
  >>>  'metadata_size': 0,
  >>>  'ref_count': 0,
  >>>  'state': 'sealed'},
  >>> ObjectID(e603ab0c92098ebf08f90bfcea33ff98f6476870): {'construct_duration': -1,
  >>>  'create_time': 1535223644,
  >>>  'data_size': 100,
  >>>  'metadata_size': 0,
  >>>  'ref_count': 1,
  >>>  'state': 'created'}}

@pcmoritz
Copy link
Contributor Author

This is now ready to give feedback or merge (I marked it as experimental so it can be evolved).

@codecov-io
Copy link

Codecov Report

❗ No coverage uploaded for pull request base (master@06e7a1b). Click here to learn what that means.
The diff coverage is 82.35%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #2470   +/-   ##
=========================================
  Coverage          ?   87.47%           
=========================================
  Files             ?      246           
  Lines             ?    43412           
  Branches          ?        0           
=========================================
  Hits              ?    37975           
  Misses            ?     5437           
  Partials          ?        0
Impacted Files Coverage Δ
cpp/src/plasma/client.h 100% <ø> (ø)
cpp/src/plasma/protocol.h 100% <ø> (ø)
cpp/src/plasma/common.h 100% <ø> (ø)
cpp/src/plasma/plasma.h 100% <ø> (ø)
cpp/src/plasma/client.cc 84.43% <100%> (ø)
python/pyarrow/tests/test_plasma.py 96.6% <100%> (ø)
cpp/src/plasma/store.cc 89.14% <50%> (ø)
cpp/src/plasma/protocol.cc 95.36% <60%> (ø)
python/pyarrow/_plasma.pyx 61.57% <95%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06e7a1b...5ff4e35. Read the comment docs.

Copy link
Contributor

@robertnishihara robertnishihara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

x, _, _ = create_object(plasma_client, 3, metadata_size=0, seal=False)
t2 = time.time()
l6 = plasma_client.list()
assert math.floor(t1) <= l6[x]["create_time"] <= math.ceil(t2)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's some chance this could be flaky due to non-monotonic clocks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'd say let's keep this clock for now, since it needs to be consistent with the timestamp from the store, where it makes more sense to use the system time than a monotonic clock. If it turns out to be flaky, we can find a different solution.

@pcmoritz pcmoritz closed this in 628b74b Aug 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants